Update int4pack related in torchchat gguf #1404
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1404
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 4 New FailuresAs of commit e7b6f14 with merge base bb72b09 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
yanbing-j
force-pushed
the
yanbing/fix_1389
branch
from
6604277 to
29d4ab7
Compare
|
I'll bump the pin real quick |
torchchat/utils/gguf_loader.py
Outdated
| ) | ||
|
|
||
| from torchao.dtypes.utils import is_device | ||
| from torchao.utils import TORCH_VERSION_AT_LEAST_2_6 |
There was a problem hiding this comment.
torchchat locks onto a specific torch version, so we don't need to check
Assume > 2.6
There was a problem hiding this comment.
The CI failures seem that torchao version is not that new, because TORCH_VERSION_AT_LEAST_2_6 is a new one. And I saw you pin pytorch nightly to 20241013, which is also not new, and this nightly does not have pytorch/pytorch#139611 inside. This is my question, because the nightly used in the CI is 20241126.
There was a problem hiding this comment.
yup, working on the bump here: #1367
We'll test your fixes on there
| weight = torch.empty( | ||
| ( | ||
| out_features, | ||
| in_features // 2, |
| out_features=out_features, | ||
| bias=False, | ||
| device="meta", | ||
| device="cpu", |
There was a problem hiding this comment.
Let's keep this as a meta device as long as we can
There was a problem hiding this comment.
Now only CPU acts different from cuda and meta. https://github.com/pytorch/torchchat/pull/1404/files/b884e295a164fa0b8cd172196e4409e51315567b#diff-28cab20c48af32e561f6e95cec7d029fa076708223a00d64afa80ad62b9b52a4R192 Use device meta here cannot tell the right shape of weight.
|
I did a quick rebase for you; feel free to change as needed |
|
Hi @Jack-Khuu , thanks for the rebase! The remaining 4 CI failures seem related to cuda device and I cannot get the obvious errors related to Int4 code change. Could you please help me find out the simple reproducer? Thanks! |
|
The cuda failures are known issues thanks for the fix |
* Update int4pack related for gguf * Update gguf_loader.py --------- Co-authored-by: Jack-Khuu <[email protected]>
3 participants
Fix pytorch/ao#1389.